Visualizing classification of natural video sequences using ! sparse, hierarchical models of cortex. !

References [1] Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture of monkey striate cortex. J. Physiol. (Lond.) 195, 215–243 (1968). [2] K. Fukushima: Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position, Biological Cybernetics, 36(4), pp. 193-202 (April 1980). [3] Riesenhuber, M. & Poggio, T. (1999). Hierarchical Models of Object Recognition in Cortex, Nature Neuroscience 2: 1019-1025. [4] T. Serre, A. Oliva and T. Poggio. A feedforward architecture accounts for rapid categorization. Proceedings of the ! National Academy of Science, 104(15), pp. 6424-6429, April 2007. [5] T. Serre, L. Wolf, S. Bileschi, M. Riesenhuber and T. Poggio. Object recognition with cortex-like mechanisms. In: IEEE Transactions on ! Pattern Analysis and Machine Intelligence, 29 (3), pp. 411-426 , 2007. [6] Paul Henning and Andrew White, Trailblazing with Roadrunner, Computing in Science & Engineering, July/August 2009, pp. 91-95. [7] Hans Moravec, When will computer hardware match the human brain?, J. Evolution &Technology, 1998. [8] S.P. Brumby, G. Kenyon, W. Landecker, C. Rasmussen, S. Swaminarayan, and L.M.A. Bettencourt, "Large-scale functional models of! visual cortex for remote sensing", Proc. 38th IEEE Applied Imagery Pattern Recognition, Vision: Humans, Animals, and Machines, 2009. [9] Pinto N, Cox DD, and DiCarlo JJ. Why is Real-World Visual Object Recognition Hard? PLoS Comp. Biology, 4(1):e27 (2008). [10] Pinto N, Cox DD, and DiCarlo JJ. A High-Throughput Screening Approach to Discovering Good Forms of Biologically-Inspired Visual Representation, GTC 2009. [11] Olshausen, B. A. and Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607-609. [12] Jim Mutch and David G. Lowe. Object class recognition and localization using sparse features with limited receptive fields. International Journal of Computer Vision (IJCV), 80(1), pp. 45-57, October 2008. [13] William K. Coulter, Cristopher J. Hillar, Guy Isley, Friedrich T. Sommer. Adaptive compressed sensing A new class of self-organizing ! coding models for neuroscience. ICASSP 2010, 5494-5497, Dallas, TX, March 2010. [14] Chang, C.-C. and C.-J. Lin (2001). LIBSVM: a library for support vector machines. (http://www.csie.ntu.edu.tw/~cjlin/libsvm)! [15] Oja, Erkki, Simplified neuron model as a principal component analyzer, Journal of Mathematical Biology 15 (3): 267–273, Nov 1982. [16] S. Brumby, L. Bettencourt, M. Ham, R. Bennett, and G. Kenyon, "Quantifying the difficulty of object recognition tasks via scaling of accuracy versus training set size", Computational and Systems Neuroscience (COSYNE) 2010, 25-28 Feb 2010, Salt Lake City, UT. [17] Will Landecker, Steven Brumby, Mick Thomure, Garrett Kenyon, Luis Bettencourt, and Melanie Mitchell, "Visualizing Classification Decisions of Hierarchical Models of Cortex", Computational and Systems Neuroscience (COSYNE) 2010, 25-28 Feb 2010, Salt Lake City, UT. [18] H. Jhuang, E. Garrote, X. Yu, V. Khilnani, T. Poggio and A. Steele and T. Serre, Automated home-cage behavioral phenotyping of mice, ! Nature Communications, 1(1), doi:10.1038/ncomms1064, 2010. [19] J. Mairal, F. Bach and J. Ponce. Task-Driven Dictionary Learning. submitted to IEEE PAMI., Sep 2010. (http://arxiv.org/abs/1009.5358) [20] S. G. Mallat and Z. Zhang, Matching Pursuits with Time-Frequency Dictionaries, IEEE Transactions on Signal Processing, pp. 3397-3415, Dec 1993. Abstract Recent work on hierarchical models of visual cortex has reported state-of-the-art accuracy on whole-scene labeling using natural still imagery. This raises the question whether the reported accuracy may be due to the sophisticated, nonbiological back-end supervised classifiers typically used (support vector machines) and/or the limited number of images used in these experiments. In particular, is ! the model classifying features from the object or the background? Previous work (Landecker, Brumby, et al., COSYNE 2010) proposed tracing the spatial support ! of a classifier’s decision back through a hierarchical cortical model to determine which parts of the image contributed to the classification, compared to the positions of objects in the scene. In this way, we can go beyond standard measures of accuracy to provide tools for visualizing and analyzing high-level object classification. We now describe new work exploring the extension of these ideas to detection of objects in video sequences of natural scenes.